Information processing apparatus, information processing method, and program

ABSTRACT

There is provided an information processing apparatus, an information processing method, and a program that are capable of detecting contact with high precision at low cost, the information processing apparatus including a contact determination unit. The contact determination unit determines contact of an operation body with a target object on the basis of a change in a normal line vector on a surface of the target object, the target object being an operation target of the operation body.

TECHNICAL FIELD

The present technology relates to an information processing apparatus,and information processing method, and a program that are capable ofdetecting contact.

BACKGROUND ART

In recent years, along with diversification of utilization forms ofinformation processing equipment, a new UI (user interface) is beingresearched. For example, a technology in which an icon is displayed on ahead-mounted display, the icon is projected on a target object (e.g.,palm and desk), and selection of the icon is input when the operationbody (e.g., finger and stylus) touches the projected icon is developed.

For contact of the operation body with the target object, a stereocamera, a 3D sensor, or the like is used. The UI recognizes the targetobject and the operation body, and detects the position of the operationbody and the plane shape of the target object on the basis of 3Dinformation acquired from the stereo camera, 3D sensor, or the like. TheUI is capable of determining, as the contact, the distance between thetarget object and the operation body being equal to or less than apredetermined value.

-   Patent Document 1: Japanese Patent Application Laid-open No.    2011-70491

SUMMARY Problem to be Solved

However, in the above-mentioned method, it needs to provide a detectionmeans such as a stereo camera and a 3D sensor for detecting contact, andthere is a problem of increase in size of the device, increase in powerconsumption, and increase in cost. Further, it is difficult to use thestereo camera and 3D sensor under sunlight because of the propertiesthereof. Further, a user's unintended input may be made in the casewhere non-contact is determined as contact when the operation bodyapproaches the target object.

In view of the circumstances as described above, it is an object of thepresent technology to provide an information processing apparatus, aninformation processing method, and a program that are capable ofdetecting contact with high precision at low cost.

Means for Solving the Problem

In order to achieve the above-mentioned object, an informationprocessing apparatus according to an embodiment of the presenttechnology includes a contact determination unit.

The contact determination unit determines contact of an operation bodywith a target object on the basis of a change in a normal line vector ona surface of the target object, the target object being an operationtarget of the operation body.

The normal line vector is a vector that faces the directionperpendicular to the surface of an object. When the operation body is incontact with the target object, the surface of the target object isdeformed by contact of the operation body and the normal line vector ischanged. Therefore, with the above-mentioned configuration, it ispossible to determine contact of the operation body with the targetobject by detecting the change in the normal line vector.

The information processing apparatus may further include an objectrecognition processing unit that recognizes the operation body and thetarget object included in a picked-up image, in which

the contact determination unit may determine contact of the operationbody with the target object on the basis of a change in a normal linevector on a surface of the target object around the operation body.

With this configuration, it is possible to reduce the amount ofcalculation for calculating the change in the normal line vector andprevent error detection because the contact determination unit uses onlyan area of the surface of the target object with which the operationbody is highly likely to be in contact for contact determination.

The contact determination unit may determine contact of the operationbody with the target object depending on an inclination of the normalline vector in a direction of the operation body.

In the case where the target object is an object (palm, etc.) in which adepression is formed by pressing of the operation body, the normal linevector is inclined in the direction of the operation body by contact ofthe operation body. Therefore, the contact determination unit is capableof using the inclination of the normal line vector in the direction ofthe operation body for contact determination.

The contact determination unit may determine contact of the operationbody with the target object depending on disturbance of an inclinationof the normal line vector.

In the case where the target object is an object (back of the hand,etc.) that gets wrinkles by pressing of the operation body, the normalline vector is disturbed by contact of the operation body, and isinclined in various directions. Therefore, the contact determinationunit is capable of using the disturbance of the inclination of thenormal line vector for contact determination.

The information processing apparatus may further include an inputcontrol unit that controls an operation input on the basis of adetermination result of the contact determination unit.

When the contact determination unit determines contact between theoperation body and the target object, the input control unit is capableof receiving it as an operation input depending on the contact positionor the number of contacts, and supplying the operation input to the OSor the like.

The information processing apparatus may further include an objectrecognition processing unit that recognizes the operation body and thetarget object included in a picked-up image, in which

the input control unit may control an operation input on the basis of ashape of the operation body recognized by the object recognitionprocessing unit and a determination result of the contact determinationunit.

The object recognition processing unit may recognize the shape of theoperation body (e.g., the shape of fingers at the time of pinchingoperation), and supply it to the input control unit. Accordingly, theinput control unit is capable of receiving the operation input dependingon the shape of the operation body.

The information processing apparatus may further include an image outputunit that generates an operation target image superimposed on the targetobject, in which

the input control unit may control an operation input on the basis of apositional relationship between the operation body and the target objectin the operation target image.

The image output unit generates an operation target image that isdisplayed on a head mounted display, projector, or the like, andsuperimposed on the target object. The input control unit is capable ofcalculating the positional relationship between the operation targetimage and the contact position on the basis of the contact position inthe operation target image and the superimposition position of theoperation target image on the target object, and receiving an operationinput depending on the content (icon, etc.) of the operation targetimage superimposed on the contact position.

The contact determination unit may determine that the operation body isin contact with the target object when an inclination of the normal linevector exceeds a threshold value.

With this configuration, it is possible to adjust the precision ofcontact detection by a threshold value.

The information processing apparatus may further include an objectrecognition processing unit that recognizes the operation body and thetarget object included in a picked-up image, in which

the contact determination unit may determine the threshold valuedepending on a kind of the target object.

With this configuration, the contact determination unit is capable ofdetermining contact with an appropriate threshold value depending on thekind of the target object. For example, it is possible to decrease thethreshold value when the target object is hard and increase thethreshold value when the target object is soft.

The contact determination unit may further determine the threshold valuedepending on a position of a surface of the target object.

For example, in the case where the target object is a palm, the degreeof depression is different for the position even if the palm is pressedwith the same pressing force. Therefore, with the above-mentionedconfiguration, it is possible to determine contact with an appropriatethreshold value depending on the position of the target object.

The contact determination unit may calculate a degree of contact of theoperation body with the target object depending on a magnitude of aninclination of the normal line vector when an inclination of the normalline vector is larger than the threshold value.

The degree of deformation of the target object due to contact of theoperation body with the target object is increased with the pressingforce of the contact. Therefore, with the above-mentioned configuration,it is possible to calculate not only the contact of the operation bodywith the target object but also the pressing force of the operation bodyon the target object.

The information processing apparatus may further include an objectrecognition processing unit that recognizes the operation body and thetarget object included in a picked-up image, in which

the contact determination unit may determine contact of the operationbody with the target object on the basis of a change in a color of thepicked-up image when the target object is a rigid body, and maydetermine contact of the operation body with the target object on thebasis of a change in the normal line vector when the target object isnot a rigid body.

In the case where the target object is a rigid body and the targetobject is not deformed by contact of the operation body, the contactdetermination unit is capable of determining contact by using the changein a color of a picked-up image. For example, because the color of afingertip is changed when the finger is pressed by an object, thecontact determination unit is capable of determining contact by usingsuch a change in a color.

The contact determination unit may determine contact of the operationbody with the target object on the basis of a change in the normal linevector when a distance between the target object and the operation bodyis equal to or less than a predetermined value.

The contact determination unit determines contact on the basis of thechange in a color of the image when the distance between the targetobject and the operation body is smaller than a predetermined value.Accordingly, it is possible to determine contact only in the case wherethe possibility of contact is high. The distance between the targetobject and the operation body can be acquired by a depth sensor thatuses a projection pattern of an infrared ray and the like to measure thedistance between the sensor and the object.

The contact determination unit may detect the normal line vector from apicked-up image picked up by polarization imaging.

The polarization imaging is achieved by a camera in which a polarizingfilter having a different polarizing direction is provided to eachpixel, and the normal line vector can be detected from the picked-upimage.

In order to achieve the above-mentioned object, a program according toan embodiment of the present technology causes an information processingapparatus to function as a contact determination unit.

The contact determination unit determines contact of an operation bodywith a target object on the basis of a change in a normal line vector ona surface of the target object, the target object being an operationtarget of the operation body.

In order to achieve the above-mentioned object, an informationprocessing method according to an embodiment of the present technologyincludes determining, by a contact determination unit, contact of anoperation body with a target object on the basis of a change in a normalline vector on a surface of the target object, the target object beingan operation target of the operation body.

Effects

As described above, according to the present technology, it is possibleto provide an information processing apparatus, an informationprocessing method, and a program that are capable of detecting contactwith high precision at low cost. It should be noted that the effectdescribed here is not necessarily limitative and may be any effectdescribed in the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 A schematic diagram showing the configuration of an informationprocessing apparatus according to a first embodiment of the presenttechnology.

FIG. 2 A schematic diagram showing an operation image superimposed on atarget object, which is generated by an image output unit of theinformation processing apparatus.

FIG. 3 A schematic diagram showing the operation image superimposed onthe target object, which is generated by the image output unit of theinformation processing apparatus.

FIG. 4 A schematic diagram showing the operation image superimposed onthe target object, which is generated by the image output unit of theinformation processing apparatus.

FIG. 5 A schematic diagram showing a hardware configuration of theinformation processing apparatus.

FIG. 6 A flowchart showing an operation of the information processingapparatus.

FIG. 7 A schematic diagram showing the normal line vector in the targetobject in the state where the operation body is not in contact with thetarget object.

FIG. 8 A schematic diagram showing the normal line vector in the targetobject in the state where the operation body is in contact with thetarget object.

FIG. 9 A schematic diagram showing the normal line vector in the targetobject in the state where the operation body is in contact with thetarget object.

FIG. 10 A flowchart showing an operation of the information processingapparatus.

FIG. 11 A flowchart showing an operation of the information processingapparatus.

FIG. 12 A flowchart showing an operation of the information processingapparatus.

FIG. 13 A schematic diagram showing the shape of the operation body.

FIG. 14 A schematic diagram showing the state of an operation input bythe operation body.

FIG. 15 A schematic diagram showing the configuration of an informationprocessing apparatus according to a second embodiment of the presenttechnology.

FIG. 16 A flowchart showing an operation of the information processingapparatus.

DESCRIPTION OF PREFERRED EMBODIMENTS First Embodiment

An information processing apparatus according to a first embodiment ofthe present technology will be described.

(Configuration of Information Processing Apparatus)

FIG. 1 is a schematic diagram showing the functional configuration of aninformation processing apparatus 100 according to a first embodiment ofthe present technology. As shown in the figure, the informationprocessing apparatus 100 includes a sound input unit 101, a soundrecognition processing unit 102, an image input unit 103, an objectrecognition processing unit 104, a contact determination unit 105, aninput control unit 106, and an image output unit 107.

As shown in the figure, the information processing apparatus 100 can beconnected to an eyewear 10. Further, the information processingapparatus 100 may be mounted on the eyewear 10. The eyewear 10 includesa display 11, a camera 12, and a microphone (not shown). The display 11can be a see-through HMD (Head Mounted Display).

The camera 12 can be a camera capable of performing polarizationimaging. Specifically, four types of polarizing films whose polarizingdirections are shifted by 90 degrees may be applied to each pixel,thereby configuring the camera 12. It is possible to acquire the normalline direction (to be described later) of the surface of the imagedobject by taking out a non-polarizing component strength, polarizationmain axis direction, and polarizing component strength by applyingFourier transform to the received light intensity of each pixel having adifferent polarizing direction, and obtaining an azimuth angle from themain axis direction and the zenith angle from the strength. Further, theinformation processing apparatus 100 may include a normal camera inaddition to the camera 12.

The sound input unit 101 is connected to the microphone of the eyewear10. The audio signal of a sound collected by the microphone is input tothe sound input unit 101. The sound input unit 101 outputs the acquiredaudio signal to the sound recognition processing unit 102.

The sound recognition processing unit 102 performs audio recognitionprocessing on the audio signal supplied from the sound input unit 101,and recognizes the operation sound by a user. In the case where theoperation sound is recognized, the sound recognition processing unit 102supplies the recognition result to the input control unit 106.

The image input unit 103 is connected to the camera 12. An image (movingimage) picked up by the camera 12 is input to the image input unit 103.The image picked up by the camera 12 is an image picked up bypolarization imaging as described above. The image input unit 103supplies the image acquired from the camera 12 (hereinafter, acquiredimage) to the object recognition processing unit 104.

The object recognition processing unit 104 checks the acquired imagesupplied from the image input unit 103 against information stored in anobject detection dictionary D to recognize an object. The objectrecognition processing unit 104 is capable of recognizing an object suchas an operation body and a target object to be described later. Theobject recognition processing unit 104 supplies the acquired image andthe object recognition result to the contact determination unit 105. Theobject detection dictionary D may be stored in the informationprocessing apparatus 100, or the object recognition processing unit 104may acquire the object detection dictionary D from a network or thelike.

The contact determination unit 105 performs contact determination on theacquired image supplied from the object recognition processing unit 104by using the object recognition result and a contact determinationthreshold value T. Details of this processing will be described later.The contact determination unit 105 supplies the determination result tothe input control unit 106. The contact determination threshold value Tmay be stored in the information processing apparatus 100, or thecontact determination unit 105 may acquire the contact determinationthreshold value T from a network or the like.

The input control unit 106 controls an operation input by a user on thebasis of the recognition result supplied from the sound recognitionprocessing unit 102 or the determination result supplied from thecontact determination unit 105. Specifically, the input control unit 106receives the contact position, pressing force, or pressing direction ofan operation body with a target object, movement of an operation bodyafter the contact, or the like, as an operation input of touching,multi-touching, dragging, pinching, or the like. The input control unit106 supplies the received operation input to the OS of the informationprocessing apparatus 100 or the like.

The image output unit 107 generates an image (moving image) displayed onthe display 11. The image output unit 107 is capable of generating anoperation image including an icon and an operation button. As describedabove, the display 11 is a see-through HMD, and the user visuallyconfirms the field of view in which videos displayed on the display 11and the real object are superimposed one another.

FIG. 2 schematically shows a user's field of view via the display 11.The user's field of view includes a palm H of the user, a finger Y ofthe user, and an operation image G displayed on the display 11. Theoperation image G includes icons P. When the user touches a part of thepalm H on which the icon P is superimposed with the finger Y, theinformation processing apparatus 100 receives selection of the icon P.

The operation image G does not necessarily need to be superimposed onthe palm. FIG. 3 is a schematic diagram showing the state where theoperation image G is superimposed on a magazine Z. As shown in thefigure, when the user touches a part of the magazine Z on which the iconP is superimposed with the finger Y, the information processingapparatus 100 receives selection of the icon P. Further, the user maydesignate the icon P with a thing other than the finger such as astylus. Hereinafter, an object (e.g., the palm H and the magazine Z) onwhich the operation image G is projected, which is designated by theuser, is referred to as the target object, and an object (e.g., thefinger Y and the stylus) that is in contact with the target object andoperates the icon P or the like is referred to as the operation body.Specifically, the information processing apparatus 100 detects contactof the operation body with the target object, and receives an operationinput in response to the contact.

The configuration of the eyewear 10 is not limited to theabove-mentioned configuration. FIG. 4 shows the state where the camera12 is provided independent of the eyewear 10. The operation image G isdisplayed on the display 11, and superimposed on the magazine Z. Thecamera 12 is provided at a position (e.g., on the upper side of themagazine Z) different from that of the eyewear 10, and is connected tothe image input unit 103.

Further, the information processing apparatus 100 may be connected to animage projection apparatus different from the eyewear 10. For example,it may be connected to a projector including a camera capable ofprojecting videos on a desk or wall surface and imaging a projectionsurface. In this case, the image output unit 107 is connected to animage projection mechanism of the projector, and projects videos on thetarget object (desk or wall). The image input unit 103 is connected tothe camera, and acquires an image including the target object and theprojected image. Alternatively, the information processing apparatus 100may be connected to an apparatus including a camera capable ofdisplaying or projecting videos on the target object and imaging thetarget object.

The above-mentioned functional configuration of the informationprocessing apparatus 100 may be connected or incorporated to/in an imageprojection apparatus such as an eyewear and a projector, or mounted on aPC, smartphone, or the like different from the image projectionapparatus. Further, the whole or a part of the functional configurationmay be on a network.

FIG. 5 is a schematic diagram showing the hardware configuration of theinformation processing apparatus 100. The above-mentioned configurationof the information processing apparatus 100 is a functionalconfiguration achieved by cooperation of the hardware configurationshown in FIG. 5 and a program.

As shown in FIG. 5, the information processing apparatus 100 includes aCPU 121, a memory 122, a storage 123, and an input/output IF 124 as ahardware configuration. These are connected to each other by a bus 125.

The CPU (Central Processing Unit) 121 controls another configuration inaccordance with a program stored in the memory 122, performs dataprocessing in accordance with the program, and stores the processingresult in the memory 122. The CPU 121 can be a microprocessor.

The memory 122 stores a program executed by the CPU 121 and data. Thememory 122 can be a RAM (Random Access Memory).

The storage 123 stores a program or data. The storage 123 can be an HDD(Hard disk drive) or an SSD (solid state drive). The storage 123 iscapable of storing the above-mentioned object detection dictionary D orthe above-mentioned contact determination threshold value T.

The input/output IF (interface) 124 receives an input to the informationprocessing apparatus 100, and supplies an output of the informationprocessing apparatus 100 to the display 11 or the like. The input/outputIF 124 includes input equipment such as a mouse and a touch panel,output equipment such as the display 11, and a connection interface suchas a network. As described above, the information processing apparatus100 is capable of receiving a user's operation input on the basis of theimage acquired by the image input unit 103. In this case, the input isreceived not via the input/output IF 124. The image generated by theimage output unit 107 is output to the display 11 via the input/outputIF 124, and displayed.

The hardware configuration of the information processing apparatus 100is not limited to those described herein as long as the above-mentionedfunctional configuration of the information processing apparatus 100 canbe achieved.

(Contact Determination Operation of Information Processing Apparatus)

An operation of the information processing apparatus 100 will bedescribed. FIG. 6 is a flowchart showing the operation of theinformation processing apparatus 100. As described above, the camera 12picks up an image, and the image input unit 103 acquires the image(acquired image). The image input unit 103 supplies the acquired imageto the object recognition processing unit 104.

The object recognition processing unit 104 recognizes the target object(St101). The object recognition processing unit 104 recognizes thetarget object by checking the acquired image acquired from the imageinput unit 103 against the object detection dictionary D. Accordingly,when the acquired image includes a palm, magazine, or the like, forexample, it is recognized as the target object.

In the case where the target object is recognized in the acquired image(St102: Yes), the image output unit 107 supplies the operation image G(see FIG. 2) to the display 11, and causes the display 11 to display theoperation image G (St103). In the case where no target object isrecognized in the acquired image (St102: No), recognition of the targetobject by the object recognition processing unit 104 (St101) isperformed again.

Next, the object recognition processing unit 104 recognizes theoperation body (St104). The object recognition processing unit 104recognizes the operation body by checking the acquired image against theobject detection dictionary D. Accordingly, when the acquired imageincludes a finger, stylus, or the like, for example, it is recognized asthe operation body. Note that the object recognition processing unit 104may recognize a plurality of operation bodies.

In the case where the operation body is recognized in the acquired image(St105: Yes), the contact determination unit 105 calculates thedirection of the normal line vector (hereinafter, normal line direction)on the surface of the target object included in the acquired image(St106). In the case where no operation body is recognized in theacquired image (St105: No), recognition of the target object by theobject recognition processing unit 104 (St101) is performed again.

FIGS. 7 and 8 are each a schematic diagram showing normal line vectors Bon the surface of the target object. FIG. 7 shows the state where theoperation body (finger Y) is not in contact with the target object (palmH). Part (a) of FIG. 7 is a plan view, and part (b) of FIG. 7 is a sideview. As shown in the figure, the contact determination unit 105 detectsthe normal line vector B on the surface of the palm H. The normal linevector is a vector perpendicular to the surface of the object, and thesize thereof is 1. As described above, the camera 12 is a camera capableof performing polarization imaging, and the contact determination unit105 is capable of detecting the normal line vector B from the acquiredimage picked up by the camera 12.

FIG. 8 shows the state where the operation body (finger Y) is in contactwith the target object (palm H). Part (a) of FIG. 8 is a plan view, andpart (b) of FIG. 8 is a side view. As shown in the figure, when thefinger Y is in contact with the palm H and presses the palm H, thesurface of the palm H is depressed. Accordingly, the normal line vectorB detected in the surface of the palm H is inclined in the direction ofthe finger Y.

The contact determination unit 105 detects contact on the basis of thechange in the normal line vector B due to contact of the operation bodywith the target object. Specifically, the contact determination unit 105is capable of determining that the finger Y is in contact with the palmH (St107: Yes) when the inclination of the direction of the normal linevector B (hereinafter, normal line direction) exceeds the contactdetermination threshold value T. The contact determination unit 105 maycompare the inclination of the normal line vector B having the largestinclination out of the normal line vectors B in which the change in thenormal line direction occurs with the contact determination thresholdvalue T, or the average value of the inclination of the normal linevector B that is inclined with the contact determination threshold valueT. Further, the contact determination unit 105 is capable of determiningthat the finger Y is not in contact with the palm H (St107: No) when theinclination of the normal line direction is smaller than the contactdetermination threshold value T.

Note that the contact determination unit 105 may determine that thefinger Y is in contact with the palm H only when the inclination of thenormal line direction in the direction of the finger Y exceeds thecontact determination threshold value T. This is because the normal linedirection may be inclined by another reason different from contact ofthe finger Y in the case where the direction is not the direction of thefinger Y even when the inclination of the normal line direction islarge. Specifically, it is possible to improve the detection precisionby using only the inclination of the normal line direction in thedirection of the finger Y for contact determination.

Further, the contact determination unit 105 may detect contact of theoperation body with the target object depending on the disturbance ofthe normal line direction. FIG. 9 is a schematic diagram showing a backof the hand K as the target object and the finger Y as the operationbody. As shown in the figure, in the case where the target object is aback of the hand, the surface is not depressed in contrast to the palmand gets wrinkles when the operation body is in contact with it. Thecontact determination unit 105 is capable of detecting such generationof wrinkles from the disturbance of the normal line direction.Specifically, the contact determination unit 105 is capable ofdetermining that the finger Y is in contact with the back of the hand Kwhen the inclination of the normal line direction in any directionexceeds the contact determination threshold value T.

The contact determination unit 105 is capable of determining whichchange in the normal line vector is used for determination by using therecognition result of the object recognition processing unit 104. Forexample, the contact determination unit 105 is capable of using theinclination of the normal line direction in the direction of theoperation body for determination in the case where the objectrecognition processing unit 104 detects an object in which a depressionis formed by pressing such as the palm H. Further, the contactdetermination unit 105 is capable of using the inclination of the normalline direction in any direction for determination in the case where theobject recognition processing unit 104 detects an object that getswrinkles by pressing such as the back of the hand K.

The contact determination threshold value T may be a constant value.Alternatively, the contact determination unit 105 may determine thecontact determination threshold value T by using the result ofrecognition of the target object by the object recognition processingunit 104. Specifically, the contact determination unit 105 is capable ofusing the contact determination threshold value T set depending on thekind of the target object. Further, the contact determination unit 105may adjust the contact determination threshold value T depending on thekind of the target object. For example, the contact determination unit105 may use a large value as the contact determination threshold value Twhen the target object is largely deformed by pressing of the operationbody, and a small value as the contact determination threshold value Twhen the target object is not largely deformed by pressing of theoperation body.

Further, the contact determination unit 105 may adjust the contactdetermination threshold value T depending on the degree of a depressionformed by pressing of the operation body on the target object, i.e., thedegree of the inclination of the normal line direction, or depending onthe contact position of the operation body with the target object. Forexample, also in the case where a finger presses a palm, a largedepression is formed in the vicinity of the center of the palm and asmall depression is formed in the vicinity of the base of the finger.Therefore, the contact determination unit 105 is capable of using anappropriate value as the contact determination threshold value T,thereby determining contact of the operation body with the target objectwith higher precision.

Note that the contact determination unit 105 does not need to detect thenormal line vector on all surfaces of the target object, and may detectonly the normal line vector on the surface of the target object aroundthe operation body recognized by the object recognition processing unit104 and use it for contact determination. Accordingly, it is possible tolimit the number of normal line vectors whose change in the normal linedirection is tracked, reduce the number of calculation, and preventerror detection.

In the case where the contact determination unit 105 determines that theoperation body is not in contact with the target object (St107: No),recognition of the target object by the object recognition processingunit 104 (St101) is performed again.

In the case where contact of the operation body with the target objectis determined, the contact determination unit 105 supplies, to the inputcontrol unit 106, the contact position with the acquired image. Theinput control unit 106 controls an operation input on the basis of theposition of the icon P displayed on the operation image G and thecontact position supplied from the contact determination unit 105. Theinput control unit 106 can regard the icon P superimposed on the contactposition as being selected, for example.

The information processing apparatus 100 performs such an operation.Since an image picked up by the camera 12 capable of performingpolarization imaging can be used to detect contact of the operation bodywith the target object as described above, it does not need to provide asensor for detecting contact in addition to the camera 12. Therefore, itis possible to reduce the size in the contact detection system and thecost. Further, because an infrared ray or the like is not used to detectcontact, it can be used outside.

Further, the information processing apparatus 100 detects contact byusing the change in the normal line vector caused due to contact of theoperation body with the target object. Therefore, the informationprocessing apparatus 100 does not detect contact when the operation bodyapproaches the target object or is slightly in contact with the targetobject, and detects contact only when the operation body is certainly incontact with the target object. Therefore, it is possible to detectcontact of the operation body with the target object with highprecision.

Note that although the case where the operation body is a finger hasbeen described above, the operation body is not limited to the fingerand may be a stylus or the like. Further, the target object is notlimited to the palm as long as the shape of the surface is changed bycontact of the operation body.

Further, the object recognition processing unit 104 checks the acquiredimage against the object detection dictionary D to recognize the object(the target object and the operation body) in the above. However, thisis not limited thereto. For example, in the case where the acquiredimage is not changed for a predetermined time and a some kind of objectenters the acquired image (i.e., the imaging range of the camera), thisobject may be recognized as the operation body.

(Contact Determination Operation of Information Processing ApparatusDepending on Kind of Target Object)

Another operation of the information processing apparatus 100 will bedescribed. FIG. 10 is a flowchart showing another operation of theinformation processing apparatus 100. Note that to the eyewear 10 (oranother apparatus) connected to the information processing apparatus100, a depth sensor capable of acquiring depth information may beprovided. The depth information is information on the distance betweenthe object and the depth sensor. As described above, the camera 12 picksup an image, and the image input unit 103 acquires the image (acquiredimage). The input unit 103 supplies the acquired image to the objectrecognition processing unit 104.

The object recognition processing unit 104 recognizes the target object(St111). The object recognition processing unit 104 is capable ofrecognizing the target object by checking the acquired image suppliedfrom the image input unit 103 against the object detection dictionary D.Accordingly, when the acquired image includes a palm, magazine, or thelike, for example, it is recognized as the target object.

In the case where the target object is recognized in the acquired image(St112: Yes), the image output unit 107 supplies the operation image G(see FIG. 2) to the display 11, and causes the display 11 to display theoperation image G (St113). In the case where no target object isrecognized in the acquired image (St112: No), recognition of the targetobject by the object recognition processing unit 104 (St111) isperformed again.

Next, the object recognition processing unit 104 recognizes theoperation body (St114). The object recognition processing unit 104recognizes the operation body by checking the acquired image against theobject detection dictionary D.

Next, the object recognition processing unit 104 determines whether ornot the target object is a rigid body (St116). The “rigid body”represents an object that is not deformed by contact of another object.Examples of the rigid body include a desk. The palm or magazine isdetermined not to be a rigid body.

The object recognition processing unit 104 determines that the targetobject is not a rigid body (St116: No), the contact determination unit105 calculates the normal line direction of the surface of the targetobject from the acquired image (St117) as described above. In the casewhere the inclination of the normal line direction exceeds the contactdetermination threshold value T (St118: Yes), the contact determinationunit 105 determines it as contact (St119). In the case where theinclination of the normal line direction is smaller than the contactdetermination threshold value T (St118: No), the contact determinationunit 105 does not determine it as contact, and recognition of the targetobject by the object recognition processing unit 104 (St111) isperformed again.

On the other hand, in the case where the object recognition processingunit 104 determines that the target object is a rigid body (St116: Yes),it is difficult to use the normal line direction for contact detectionbecause the normal line direction is not changed even by contact of theoperation body with the target object. Therefore, the contactdetermination unit 105 can determine contact without using the normalline direction.

Specifically, the contact determination unit 105 calculates the distancebetween the target object and the operation body (St120). The distancebetween the target object and the operation body can be calculated byusing the depth information supplied from the depth sensor. In the casewhere the distance between the target object and the operation body isequal to or less than a constant value (St121: Yes), the contactdetermination unit 105 determines it as contact (St119). In the casewhere the distance between the target object and the operation body isnot equal to or less than the constant value (St121: No), recognition ofthe target object by the object recognition processing unit 104 (St111)is performed again.

The information processing apparatus 100 is capable of performing alsosuch an operation. Even if the target object is a rigid body and it isdifficult to perform contact determination using the normal linedirection, it is possible to perform contact determination. Note thatthe contact determination unit 105 detects contact of the operation bodywith the target object by using depth information in the case where thetarget object is a rigid body in the above description, another methodmay be used to detect contact.

For example, the color of a fingertip is changed when the finger ispressed by an object. In the case where the finger is not in contactwith an object, for example, the color of a fingertip viewed through anail is pink. However, the color of the tip of the finger is changed towhite when the finger is pressed by an object. The contact determinationunit 105 is capable of detecting contact of the finger with the targetobject by comparing such a change in the color of a fingertip with apattern set in advance. Further, the contact determination unit 105 iscapable of detecting contact on the basis of the change in a colorsimilarly to the color of the fingertip as long as the operation body isan object whose color is changed when the object is pressed by thetarget object.

(Contact Determination Operation of Information Processing ApparatusCombined by Use of Depth Information)

Another operation of the information processing apparatus 100 will bedescribed. Note that a depth sensor capable of acquiring depthinformation is provided to the eyewear 10 (or another apparatus)connected to the information processing apparatus 100.

FIG. 11 is a flowchart showing another operation of the informationprocessing apparatus 100. As described above, the camera 12 picks up animage, and the image (acquired image) is acquired by the image inputunit 103. The image input unit 103 supplies the acquired image to theobject recognition processing unit 104.

The object recognition processing unit 104 recognizes the target object(St131). The object recognition processing unit 104 recognizes thetarget object by checking the acquired image acquired from the imageinput unit 103 against the object detection dictionary D.

In the case where the target object is recognized in the acquired image(St132: Yes), the image output unit 107 supplies the operation image G(see FIG. 2) to the display 11, and causes the display 11 to display theoperation image G (St133). In the case where no target object isrecognized in the acquired image (St132: No), recognition of the targetobject by the object recognition processing unit 104 (St131) isperformed again.

Next, the object recognition processing unit 104 recognizes theoperation body (St134). The object recognition processing unit 104 iscapable of recognizing the operation body by checking the acquired imageagainst the object detection dictionary D. Note that the objectrecognition processing unit 104 may recognize a plurality of operationbodies.

In the case where the operation body is recognized in the acquired image(St135: Yes), the contact determination unit 105 calculates the distancebetween the operation body and the target object by using the depthinformation (St136). The contact determination unit 105 is capable ofacquiring the depth information from the depth sensor, as describedabove. In the case where no operation body is recognized in the acquiredimage (St135: No), recognition of the target object by the objectrecognition processing unit 104 (St131) is performed again.

In the case where the distance between the operation body and the targetobject is equal to or less than a predetermined value set in advance(St137: Yes), the contact determination unit 105 detects the normal linevector on the surface of the target object, and calculates the normalline direction (St138). In the case where the distance between theoperation body and the target object is larger than the predeterminedvalue (St137: No), recognition of the target object by the objectrecognition processing unit 104 (St131) is performed again.

Thereafter, the contact determination unit 105 compares the inclinationof the normal line direction with the contact determination thresholdvalue T, and determines that the operation body is in contact with thetarget object (St140) when the inclination of the normal line directionexceeds the contact determination threshold value T (St139: Yes) asdescribed above. In the case where the inclination of the normal linedirection is smaller than the contact determination threshold value T(St139: No), recognition of the target object by the object recognitionprocessing unit 104 (St131) is performed again.

In the case where contact of the operation body with the target objectis determined, the contact determination unit 105 supplies, to the inputcontrol unit 106, the contact position with the acquired image. Theinput control unit 106 controls an operation input on the basis of theposition of the icon P displayed on the operation image G and thecontact position supplied from the contact determination unit 105. Theinput control unit 106 can regard the icon P superimposed on the contactposition as being selected, for example.

The information processing apparatus 100 is capable of performing alsosuch an operation. By using the depth information, the informationprocessing apparatus 100 is capable of acquiring the distance betweenthe operation body and the target object, and performing contactdetermination by using the change in the normal line vector only in thecase where the distance between the operation body and the target objectis sufficiently small and the possibility of contact is high.

(Contact and Operation Determination Operations of InformationProcessing Apparatus)

The Information processing apparatus 100 is capable of determining thetype (dragging, pinching, etc.) of a contact operation as well ascontact between the operation body and the target object. FIG. 12 is aflowchart showing contact and operation determination operations of theinformation processing apparatus 100.

The flow of the step of contact determination (St151) is the same asthat of the contact determination operation shown in FIG. 6. Further,the flow of the same step (St151) may be the same as that of the contactdetermination operation depending on the kind of the target object shownin FIG. 10 or the contact reaction operation combined by the usage ofdepth information shown in FIG. 11.

In the case where contact is determined in the contact determinationstep (St152: Yes), the contact determination unit 105 determines thedegree of contact (pressing) of the operation body with the targetobject (St153). The contact determination unit 105 is capable ofobtaining the degree of contact from the magnitude of the inclinationwhen the inclination of the normal line direction exceeds the contactdetermination threshold value T. In the case where no contact isdetermined in the contact determination step (St152: No), the step ofcontact determination (St151) is repeated.

The contact determination unit 105 supplies, to the input control unit106, the contact position with the acquired image and the degree ofpressing force of the operation body on the target object. The inputcontrol unit 106 determines whether or not the contact position betweenthe operation body and the target object is changed (St154). In the casewhere the contact position is changed (St154: Yes), the input controlunit 106 determines whether or not the number of operation bodies is two(St155). In the case where the contact position is not changed (St155:No), the step of contact determination (St151) is performed again.

In the case where the number of operation bodies is two (St155: Yes),the input control unit 106 determines that the operation by theoperation body is pinching. On the other hand, in the case where thenumber of operation bodies is not two, the input control unit 106determines that the operation by the operation body is dragging. Notethat in the case where the number of operation bodies is two, the inputcontrol unit 106 may determine it as dragging (multi-touch dragging) bya plurality of operation bodies in the case where the movementdirections of the contact places are substantially in parallel with eachother. Further, in the case where the operation by the operation body ispinching, the input control unit 106 may determine that the operation ispinching in or pinching out depending on the movement direction of thecontact place.

At this time, the input control unit 106 may determine the type of theoperation by using the object recognition result acquired from theobject recognition processing unit 104. FIG. 13 is a schematic diagramshowing the shape of a finger as the operation body. In the case where aparticular shape of the operation body (e.g., orientation of the finger)is registered in the object recognition dictionary D in advance, theobject recognition processing unit 104 supplies the information to theinput control unit 106. The input control unit 106 is capable of usingthe shape of the operation body for determination of the type ofoperation. For example, in the example shown in FIG. 13, the objectrecognition processing unit 104 can recognize that the shape of thefingers is a shape for the pinching operation (U-shape formed by thethumb and the index finger), and the input control unit 106 candetermine that the operation is a pinching operation only in the casewhere the shape of the fingers is a shape for the pinching operation.

The input control unit 106 controls an operation input depending on thetype of the operation by the operation body. At this time, the inputcontrol unit 106 may change the degree of effects (cursor movementspeed, a scroll speed, etc.) by dragging depending on the degree of thepressing force of the operation body on the target object.

FIG. 14 shows an example of the UI of a music reproduction applicationby the above-mentioned operation of the information processing apparatus100. As shown in the figure, the information processing apparatus 100 iscapable of selecting the sound volume, music to be reproduced, or thelike, depending on the direction or degree of pressing of the operationbody on the target object.

Second Embodiment

An information processing apparatus according to a second embodiment ofthe present technology will be described.

(Configuration of Information Processing Apparatus)

FIG. 15 is a schematic diagram showing the functional configuration ofan information processing apparatus 200 according to the secondembodiment of the present technology. As shown in the figure, theinformation processing apparatus 200 includes a sound input unit 201, asound recognition processing unit 202, an image input unit 203, acontact determination unit 204, an input control unit 205, and an imageoutput unit 206. The information processing apparatus 200 according tothis embodiment includes no object recognition processing unit.

As shown in the figure, the information processing apparatus 200 can beconnected to the eyewear 10. Because the configuration of the eyewear 10is similar to that in the first embodiment, description thereof will beomitted.

The sound input unit 201 is connected to the microphone of the eyewear10. The audio signal of a sound collected by the microphone is input tothe sound input unit 201. The sound input unit 201 outputs the acquiredaudio signal to the sound recognition processing unit 202.

The sound recognition processing unit 202 performs audio recognitionprocessing on the audio signal supplied from the sound input unit 201,and recognizes the operation sound by a user. In the case where theoperation sound is recognized, the sound recognition processing unit 202supplies the audio recognition result to the input control unit 205.

The image input unit 203 is connected to the camera 12. An image (movingimage) picked up by the camera 12 is input to the image input unit 203.The image input unit 203 supplies the image acquired from the camera 12(hereinafter, acquired image) to the contact determination unit 204.

The contact determination unit 204 performs contact determination on theacquired image supplied from the image input unit 203 by using thecontact determination threshold value T. Details of this processing willbe described later. The contact determination unit 204 supplies thedetermination result to the input control unit 205. The contactdetermination threshold value T may be stored in the informationprocessing apparatus 200, or the contact determination unit 204 mayacquire the contact determination threshold value T from a network orthe like.

The input control unit 205 controls an operation input by a user on thebasis of the recognition result supplied from the sound recognitionprocessing unit 202 or the determination result supplied from thecontact determination unit 204. Specifically, the input control unit 205receives the contact position, pressing force, or pressing direction ofan operation body with a target object, movement of an operation bodyafter the contact, or the like, as an operation input of touching,multi-touching, dragging, pinching, or the like. The input control unit205 supplies the received operation input to the OS of the informationprocessing apparatus 200 or the like.

The image output unit 206 generates an image (moving image) displayed onthe display 11. The image output unit 206 is capable of generating anoperation image including an icon and an operation button. Similarly tothe first embodiment, the user visually confirms the field of view inwhich videos displayed on the display 11 and the real object aresuperimposed one another (see FIG. 2 and FIG. 3).

The information processing apparatus 200 may be connected to anapparatus including a camera capable of displaying or projecting videoson the target object and imaging the target object, in addition to theeyewear 10.

The above-mentioned functional configuration of the informationprocessing apparatus 200 may be connected or incorporated to/in an imageprojection apparatus such as an eyewear and a projector, or mounted on aPC, smartphone, or the like different from the image projectionapparatus. Further, the whole or a part of the functional configurationmay be on a network.

The above-mentioned functional configuration of the informationprocessing apparatus 200 can be achieved by the hardware configurationshown in the first embodiment.

(Contact Determination Operation of Information Processing Apparatus)

An operation of the information processing apparatus 200 will bedescribed. FIG. 16 is a flowchart showing the operation of theinformation processing apparatus 200. As described above, the camera 12picks up an image, and the image (acquired image) is acquired by theimage input unit 203. The image input unit 203 supplies the acquiredimage to the contact determination unit 204.

The image output unit 206 supplies the operation image G (see FIG. 2) tothe display 11, and causes the display 11 to display the operation imageG (St201). In contrast to the first embodiment, recognition processingof the target object is not performed. However, the user is capable ofaiming the display 11 at the target object (desk or wall surface),thereby projecting the operation image G on the target object.

The contact determination unit 204

The contact determination unit 204 compares the inclination of thenormal line direction with the contact determination threshold value T,and determines that the operation body is in contact with the targetobject (St204) when the change in the normal line direction exceeds thecontact determination threshold value T (St203: Yes). In the case wherethe change in the normal line direction is smaller than the contactdetermination threshold value T (St203: No), calculation of the normalline direction (St202) is performed again.

In the case where contact of the operation body with the target objectis determined, the contact determination unit 204 supplies, to the inputcontrol unit 205, the contact position with the acquired image. Theinput control unit 205 controls an operation input on the basis of thecontact determination result supplied from the contact determinationunit 204.

The information processing apparatus 200 performs such an operation.Since an image picked up by the camera 12 capable of performingpolarization imaging can be used to detect contact of the operation bodywith the target object as described above, it does not need to provide asensor for detecting contact in addition to the camera 12. Therefore, itis possible to reduce the size in the contact detection system and thecost. Further, because an infrared ray or the like is not used to detectcontact, it can be used outside.

Further, the information processing apparatus 200 detects contact byusing the change in the normal line direction caused due to contact ofthe operation body with the target object. Therefore, the informationprocessing apparatus 100 does not detect contact when the operation bodyapproaches the target object or is slightly in contact with the targetobject, and detects contact only when the operation body is certainly incontact with the target object. Therefore, it is possible to detectcontact of the operation body with the target object with highprecision.

The information processing apparatus 200 performs such an operation.Note that the information processing apparatus 200 may perform contactdetermination (see FIG. 11) by using the change in a color and depthinformation, similarly to the first embodiment. Further, the informationprocessing apparatus 200 may determine the type of the operation bycontact after the contact determination (see FIG. 12).

Note that although the case where the operation body is a finger hasbeen described in the above, the operation body is not limited to afinger and may be a stylus or the like. Further, also the target objectis not limited to a palm as long as the shape of the surface thereof ischanged by contact of the operation body with the target object.

It should be noted that the present technology may take the followingconfigurations.

(1)

An information processing apparatus, including

a contact determination unit that determines contact of an operationbody with a target object on the basis of a change in a normal linevector on a surface of the target object, the target object being anoperation target of the operation body.

(2)

The information processing apparatus according to (1) above, furtherincluding

an object recognition processing unit that recognizes the operation bodyand the target object included in a picked-up image, in which

the contact determination unit determines contact of the operation bodywith the target object on the basis of a change in a normal line vectoron a surface of the target object around the operation body.

(3)

The information processing apparatus according to (1) or (2) above, inwhich

the contact determination unit determines contact of the operation bodywith the target object depending on an inclination of the normal linevector in a direction of the operation body.

(4)

The information processing apparatus according to any one of (1) to (3)above, in which

the contact determination unit determines contact of the operation bodywith the target object depending on disturbance of an inclination of thenormal line vector.

(5)

The information processing apparatus according to any one of (1) to (4)above, further including

an input control unit that controls an operation input on the basis of adetermination result of the contact determination unit.

(6)

The information processing apparatus according to any one of (1) to (5)above, further including

an object recognition processing unit that recognizes the operation bodyand the target object included in a picked-up image, in which

the input control unit controls an operation input on the basis of ashape of the operation body recognized by the object recognitionprocessing unit and a determination result of the contact determinationunit.

(7)

The information processing apparatus according to any one of (1) to (6)above, further including

an image output unit that generates an operation target imagesuperimposed on the target object, in which

the input control unit controls an operation input on the basis of apositional relationship between the operation body and the target objectin the operation target image.

(8)

The information processing apparatus according to any one of (1) to (7)above, in which

the contact determination unit determines that the operation body is incontact with the target object when an inclination of the normal linevector exceeds a threshold value.

(9)

The information processing apparatus according to any one of (1) to (8)above, further including

an object recognition processing unit that recognizes the operation bodyand the target object included in a picked-up image, in which

the contact determination unit determines the threshold value dependingon a kind of the target object.

(10)

The information processing apparatus according to any one of (1) to (9)above, in which

the contact determination unit further determines the threshold valuedepending on a position of a surface of the target object.

(11)

The information processing apparatus according to any one of (1) to (10)above, in which

the contact determination unit calculates a degree of contact of theoperation body with the target object depending on a magnitude of aninclination of the normal line vector when an inclination of the normalline vector is larger than the threshold value.

(12)

The information processing apparatus according to any one of (1) to (11)above, further including

an object recognition processing unit that recognizes the operation bodyand the target object included in a picked-up image, in which

the contact determination unit determines contact of the operation bodywith the target object on the basis of a change in a color of thepicked-up image when the target object is a rigid body, and determinescontact of the operation body with the target object on the basis of achange in the normal line vector when the target object is not a rigidbody.

(13)

The information processing apparatus according to any one of (1) to (12)above, in which

the contact determination unit determines contact of the operation bodywith the target object on the basis of a change in the normal linevector when a distance between the target object and the operation bodyis equal to or less than a predetermined value.

(14)

The information processing apparatus according to any one of (1) to (13)above, in which

the contact determination unit detects the normal line vector from apicked-up image picked up by polarization imaging.

(15)

A program that causes an information processing apparatus to function as

a contact determination unit that determines contact of an operationbody with a target object on the basis of a change in a normal linevector on a surface of the target object, the target object being anoperation target of the operation body.

(16)

An information processing method, including

determining, by a contact determination unit, contact of an operationbody with a target object on the basis of a change in a normal linevector on a surface of the target object, the target object being anoperation target of the operation body.

DESCRIPTION OF REFERENCE NUMERALS

-   -   100, 200 information processing apparatus    -   101, 201 sound input unit    -   102, 202 audio recognition processing unit    -   103, 203 image input unit    -   104 object recognition processing unit    -   105, 204 contact determination unit    -   106, 205 input control unit    -   107, 206 image output unit

1. An information processing apparatus, comprising a contactdetermination unit that determines contact of an operation body with atarget object on the basis of a change in a normal line vector on asurface of the target object, the target object being an operationtarget of the operation body.
 2. The information processing apparatusaccording to claim 1, further comprising an object recognitionprocessing unit that recognizes the operation body and the target objectincluded in a picked-up image, wherein the contact determination unitdetermines contact of the operation body with the target object on thebasis of a change in a normal line vector on a surface of the targetobject around the operation body.
 3. The information processingapparatus according to claim 1, wherein the contact determination unitdetermines contact of the operation body with the target objectdepending on an inclination of the normal line vector in a direction ofthe operation body.
 4. The information processing apparatus according toclaim 1, wherein the contact determination unit determines contact ofthe operation body with the target object depending on disturbance of aninclination of the normal line vector.
 5. The information processingapparatus according to claim 1, further comprising an input control unitthat controls an operation input on the basis of a determination resultof the contact determination unit.
 6. The information processingapparatus according to claim 5, further comprising an object recognitionprocessing unit that recognizes the operation body and the target objectincluded in a picked-up image, wherein the input control unit controlsan operation input on the basis of a shape of the operation bodyrecognized by the object recognition processing unit and a determinationresult of the contact determination unit.
 7. The information processingapparatus according to claim 5, further comprising an image output unitthat generates an operation target image superimposed on the targetobject, wherein the input control unit controls an operation input onthe basis of a positional relationship between the operation body andthe target object in the operation target image.
 8. The informationprocessing apparatus according to claim 1, wherein the contactdetermination unit determines that the operation body is in contact withthe target object when an inclination of the normal line vector exceedsa threshold value.
 9. The information processing apparatus according toclaim 8, further comprising an object recognition processing unit thatrecognizes the operation body and the target object included in apicked-up image, wherein the contact determination unit determines thethreshold value depending on a kind of the target object.
 10. Theinformation processing apparatus according to claim 9, wherein thecontact determination unit further determines the threshold valuedepending on a position of a surface of the target object.
 11. Theinformation processing apparatus according to claim 8, wherein thecontact determination unit calculates a degree of contact of theoperation body with the target object depending on a magnitude of aninclination of the normal line vector when an inclination of the normalline vector is larger than the threshold value.
 12. The informationprocessing apparatus according to claim 1, further comprising an objectrecognition processing unit that recognizes the operation body and thetarget object included in a picked-up image, wherein the contactdetermination unit determines contact of the operation body with thetarget object on the basis of a change in a color of the picked-up imagewhen the target object is a rigid body, and determines contact of theoperation body with the target object on the basis of a change in thenormal line vector when the target object is not a rigid body.
 13. Theinformation processing apparatus according to claim 1, wherein thecontact determination unit determines contact of the operation body withthe target object on the basis of a change in the normal line vectorwhen a distance between the target object and the operation body isequal to or less than a predetermined value.
 14. The informationprocessing apparatus according to claim 1, wherein the contactdetermination unit detects the normal line vector from a picked-up imagepicked up by polarization imaging.
 15. A program that causes aninformation processing apparatus to function as a contact determinationunit that determines contact of an operation body with a target objecton the basis of a change in a normal line vector on a surface of thetarget object, the target object being an operation target of theoperation body.
 16. An information processing method, comprisingdetermining, by a contact determination unit, contact of an operationbody with a target object on the basis of a change in a normal linevector on a surface of the target object, the target object being anoperation target of the operation body.